Traffic Light Control by Multiagent Reinforcement Learning Systems

Traffic light control is one of the main means of controlling road traffic. Improving traffic control is important because it can lead to higher traffic throughput and reduced congestion. This chapter describes multiagent reinforcement learning techniques for automatic optimization of traffic light controllers. Such techniques are attractive because they can automatically discover efficient control strategies for complex tasks, such as traffic control, for which it is hard or impossible to compute optimal solutions directly and hard to develop hand-coded solutions. First the general multi-agent reinforcement learning framework is described that is used to control traffic lights in this work. In this framework, multiple local controllers (agents) are each responsible for the optimization of traffic lights around a single traffic junction, making use of locally perceived traffic state information (sensed cars on the road), a learned probabilistic model of car behavior, and a learned value function which indicates how traffic light decisions affect long-term utility, in terms of the average waiting time of cars. Next, three extensions are described which improve upon the basic framework in various ways: agents (traffic junction controllers) taking into account congestion information from neighboring agents; handling partial observability of traffic states; and coordinating the behavior of multiple agents by coordination graphs and the max-plus algorithm. Bram Bakker Informatics Institute, University of Amsterdam, Science Park 107, 1098 XG Amsterdam, The Netherlands, e-mail: [email protected] Shimon Whiteson Informatics Institute, University of Amsterdam, Science Park 107, 1098 XG Amsterdam, The Netherlands e-mail: [email protected] Leon J.H.M. Kester Integrated Systems, TNO Defense, Safety and Security, Oude Waalsdorperweg 63, 2597 AK Den Haag, The Netherlands e-mail: [email protected] Frans C.A. Groen Informatics Institute, University of Amsterdam, Science Park 107, 1098 XG Amsterdam, The Netherlands e-mail: [email protected]